Effectiveness of chatbots on COVID vaccine confidence and acceptance in Thailand, Hong Kong, and Singapore

Chatbots have become an increasingly popular tool in the field of health services and communications. Despite chatbots’ significance amid the COVID-19 pandemic, few studies have performed a rigorous evaluation of the effectiveness of chatbots in improving vaccine confidence and acceptance. In Thailand, Hong Kong, and Singapore, from February 11th to June 30th, 2022, we conducted multisite randomised controlled trials (RCT) on 2,045 adult guardians of children and seniors who were unvaccinated or had delayed vaccinations. After a week of using COVID-19 vaccine chatbots, the differences in vaccine confidence and acceptance were compared between the intervention and control groups. Compared to non-users, fewer chatbot users reported decreased confidence in vaccine effectiveness in the Thailand child group [Intervention: 4.3 % vs. Control: 17%, P = 0.023]. However, more chatbot users reported decreased vaccine acceptance [26% vs. 12%, P = 0.028] in Hong Kong child group and decreased vaccine confidence in safety [29% vs. 10%, P = 0.041] in Singapore child group. There was no statistically significant change in vaccine confidence or acceptance in the Hong Kong senior group. Employing the RE-AIM framework, process evaluation indicated strong acceptance and implementation support for vaccine chatbots from stakeholders, with high levels of sustainability and scalability. This multisite, parallel RCT study on vaccine chatbots found mixed success in improving vaccine confidence and acceptance among unvaccinated Asian subpopulations. Further studies that link chatbot usage and real-world vaccine uptake are needed to augment evidence for employing vaccine chatbots to advance vaccine confidence and acceptance.


INTRODUCTION
As of January 2022, >500 million confirmed cases of COVID-19 have been reported worldwide 1 . COVID-19 vaccines have been proven to be effective in lowering hospitalisation and mortality rates 2 , and are considered the most efficient tools for reducing severe disease and economic burden 3,4 . However, by January 28th, 2022, only 17 countries worldwide had attained a vaccination coverage of 70 percent 4 .
Many factors contribute to low COVID-19 vaccination coverage, including vaccine supply and distribution, access to healthcare facilities, and vaccine hesitancy. In particular, vaccine hesitancy, named one of the ten biggest threats to global health by the World Health Organization (WHO) 5 , refers to the "motivational state of being conflicted about, or opposed to, getting vaccinated." 6 Vaccine hesitancy is a major obstacle to increasing vaccine uptake and returning to normalcy, especially among seniors and parents of children [7][8][9][10] . As of January 28th, 2022, immediately preceding the study period, the percentage of elderly people aged 60 and above who had received at least one dose of vaccine was 75.1% in Thailand and 45% in Hong Kong; vaccination coverage of at least one dose among children was 34.5% in Hong Kong and 44% in Singapore, while Thailand had not yet begun vaccinating children [11][12][13][14] .
Factors affecting COVID-19 vaccine hesitancy included perceptions of vaccine importance, efficacy and safety, concerns about side effects, vaccine accessibility, and personal or religious beliefs [15][16][17][18] . Major concerns from seniors regarded the risk of serious adverse events following immunisation, such as deaths and complications due to old age and medical history 19 . For example, in Thailand, vaccines were provided by both the government and the private sector; however, results from crosssectional surveys indicated that vaccination uptake among Thai people, especially among seniors, was low compared to other Southeast Asian countries 20 . Further, as children became newly eligible for vaccinations, there were concerns over the necessity or safety of child vaccinations. A large-scale study showed that Thai parents' willingness to get their children vaccinated was the lowest among lower-and middle-income countries, due to uncertainty about COVID-19 vaccines' effectiveness and safety 21 . Likewise, Hong Kong parents had doubts about vaccine safety for their children, and Hong Kong caregivers' willingness to have their children vaccinated was lower compared to other countries 22 . In Singapore, some parents voiced concerns about the risk of contracting COVID-19 despite taking their children to get vaccinated 23 . Studies showed that low vaccine uptake in the region could be linked to high complacency from relatively low 1 numbers of daily COVID-19 cases and low confidence in vaccine safety, coupled with a lack of knowledge about differences between each vaccine type 10,20-24 . Compared to previous infectious disease outbreaks, a significant challenge during the COVID-19 pandemic was the overabundance of online and offline information, which included deliberate attempts to undermine and jeopardize global pandemic response measures by disseminating misinformation 25 . In response, the need for timely vaccination communication called for more effective use of social media and digital technologies such as machine learning, artificial intelligence, and conversation technology 26 . Among different digital interventions, chatbots have become an increasingly popular tool in health communication and services due to their ubiquitous access points and potential for massive information dissemination. There are many possible benefits to chatbots in an outbreak setting, including the relatively limited resources required to employ them after development, offering accessibility to clear, accurate, and timely information as outbreaks evolve, and freeing up the time of healthcare workers' to address more complex issues 27 . However, as the use of chatbots in the context of health communication, especially in vaccine communication, is a novel approach, rigorous evaluations of their impact and potential use cases are very limited 28 .
In this study, we test the effectiveness of COVID-19 chatbots on people who were unvaccinated or had delayed vaccinations until the government vaccine mandates in Thailand, Hong Kong, and Singapore. Among many factors affecting vaccine attitudes and behaviours, we focus on participants' changes in levels of vaccine confidence and acceptance, before and after using the COVID-19 chatbots. Given differentiating findings per target populations, we encourage future studies to further substantiate the evidence of vaccine chatbots' effectiveness in advancing vaccine confidence and acceptance.

Recruitment and retention
From February 11th, 2022 to June 30th, 2022, 2045 participants were enrolled and randomly assigned to the control and intervention groups. After excluding 1,280 participants who were lost to follow-up, responses from 748 participants were included in the final analysis. (Fig. 1 and Supplementary Fig. 8).

Participant characteristics
Across all five study groups in the three locations, the baseline characteristics between the control and intervention groups were mostly comparable (Supplementary Tables 1 and 2). We noted an overrepresentation of minority subpopulation (i.e., Filipinos and Indonesians) in the Hong Kong and Singapore groups.

Findings from the randomised controlled trial
In the Thailand child group, results from a one-way comparison using Fisher's exact test showed that participants in the intervention group were less likely to decrease in confidence regarding vaccine effectiveness: "vaccines are effective" [Intervention: 4.3% vs. Control: 17%, P = 0.023], "vaccines are effective regardless of manufacturers" [14% vs. 55%, P < 0.001], and "vaccines are effective against all variants" [8.7% vs. 48%, P < 0.001] compared to the control group (Table 1). In the Hong Kong child group, we did not detect any statistically significant findings regarding vaccine confidence. However, participants in the intervention group were more likely to report decrease in vaccine acceptance ("has your child received/do you intend for your child to receive a COVID-19 vaccine?") [26% vs. 12%, P = 0.028] ( Table 1). In the Singapore child group, we found more participants in the intervention group with decreased vaccine confidence regarding safety "vaccines are safe" [29% vs. 10%, P = 0.041]. Regarding secondary outcomes, the Fisher's exact test showed that fewer participants in the Thailand intervention group reported decreased perceived need of COVID-19 vaccines ("my child does not need to be vaccinated" [13% vs. 32%, P = 0.006]) and perceived risk of COVID-19 disease ("COVID-19 is a serious disease") [7.2% vs. 25%, P = 0.008] compared to the control group, after the intervention (Supplementary Table 3 Table 9). In the Thailand senior group, a comparison using Fisher's exact test showed that more participants in the intervention group reported a decrease in vaccine confidence related to effectiveness in reducing severe conditions [12% vs. 21%, P = 0.024] and misinformation awareness related to COVID-19 vaccines' completion of clinical trials [3.4% vs.18%, P = 0.026] than the control group (Table 2 and Supplementary Table 4). In the Hong Kong senior group, Fisher's exact test showed no significant findings on either the primary or secondary outcomes (Supplementary Table  4). Similar results were obtained from sensitivity analyses excluding Filipino and Indonesian subgroups (Supplementary  Tables 5-8 and Supplementary Figs. 2-3).

DISCUSSION
This study reports the impact of COVID-19 chatbots on vaccine confidence and acceptance of individuals who are unvaccinated or have delayed vaccinations in Thailand, Hong Kong, and Singapore. Most notably, in the Thai child group, we saw greater improvements in the chatbot users' beliefs regarding vaccine effectiveness and debunking misinformation about COVID-19 vaccines and infertility. However, we also observed differentiating results based on the target population and region including backfire effects of chatbot use on both vaccine acceptance and confidence. The impact of chatbots was more effective in improving vaccine confidence and acceptance among unsure minorities and people with lower level of education, but less so for the highly educated population. This study is a multisite, parallel RCT on multilingual vaccine chatbots. It was conducted in three Asian regions, one being upper-middle-income and two being high-income. As this study was conducted during the aggressive implementation of containment interventions such as social distancing rules and mandatory vaccine pass schemes by the governments in our study sites, we employed the RCT design to evaluate the impact of the chatbot intervention. Chatbot development and evaluation were constantly updated and tailored to changing local epidemic situations and vaccine policies and programmes (e.g., approval of the 5-11 age group vaccinations) 22   variables of interest. The chatbots' high practicality, flexibility (i.e., the ability to adapt to different settings, such as HPV vaccination campaigns), and scalability demonstrated promising evidence for future research and applications. Nevertheless, our study has several limitations. First, our sample sizes across all three regions were smaller than our target sample sizes. A combination of heightened risk perception owing to increased daily case counts and public health measures, such as school-based vaccine rollout and vaccine mandates, led to surges in vaccination uptake in our study locations, leaving only a small population who remained unvaccinated during our study period. Although we have found some statistically significant findings, our small sample sizes might lack sufficient power to detect the effect of the intervention in its entirety, e.g., dose-dependent effect (Supplementary Table 14). In order to recruit more participants, we relaxed our study criteria to include those who delayed their vaccination until the implementation of governmental vaccine mandates. Another possible reason for our small sample size is the high proportions of participants lost to follow-up, potentially due to our chatbot's design 31 . For instance, our chatbot was not able to recognize users' emotions and tailor phrase responses to questions. In addition, since participants recruited by Premise were more familiar with surveys related to market research rather than vaccines, their indifference to domains of chatbot contents might have led to user dissatisfaction and consequently a high drop-out rate. Second, our sample population was not representative of the populations in respective regions. To address this limitation, we applied population weightings in regression models based on respective regions' census data to adjust for potential biases. Third, the chatbot employed in Hong Kong and Singapore only had COVID-19 vaccine-related content and was unable to answer general COVID-19 questions (i.e., COVID-19 home care instructions, daily COVID-19 cases). As a result, participants might have engaged less with the chatbot and rated the chatbots as less helpful than they would have otherwise. Fourth, our study might have social desirability bias since outcomes are self-reported amid active governmental encouragement and mandates on vaccination during the Omicron outbreak. Fifth, our study design incorporated responses of guardians to gauge vaccine confidence and acceptance of unvaccinated seniors due to lacking eligible senior participants in the existing panel. It is inferred that changes in the guardians' vaccine confidence and acceptance will affect those of unvaccinated seniors, as family support is one of the most influential factors in senior vaccine hesitancy in this region [32][33][34] . However, this proxy approach might not directly reflect the changes in unvaccinated seniors' vaccine confidence and acceptance. Further studies could advance the generalizability of chatbot interventions to target seniors, instead of their guardians, directly, and also investigate whether improve confidence in vaccine effectiveness could be translated into vaccination actions. Finally, our study focused on vaccine confidence and acceptance among numerous other factors that could drive vaccine hesitancy and deter children and seniors' COVID-19 vaccine uptake. Studies suggest that differences in sociodemographic structures, health literacy, prevalence of chronic diseases, distributions of vaccine supplies, convenience to vaccinations, trust towards healthcare systems, governments, and vaccine developers impact vaccination coverages [35][36][37][38][39][40] . While vaccine confidence and acceptance are important attributes of vaccine uptake, our findings are not to be interpreted as the sole indicator of vaccination behaviours.
The first major finding of this study is that there was an increase in vaccine acceptance and confidence among some chatbot users across different study groups, adding to a growing body of evidence that well-designed and implemented chatbots can have a positive influence on health behaviours 19,27,[41][42][43][44][45][46][47] . While there has been growing interest in chatbots across a range of public health areas 19,27,[41][42][43]48 , very few studies have previously investigated the effectiveness of chatbots in promoting vaccine acceptance using RCTs 43,[49][50][51] . For COVID-19 vaccination, our study lends weight to previous findings that interactive conversations between chatbots and users can contribute to increased vaccine confidence, as seen in the Thailand child group 27,52 . Additionally, since our chatbots were hosted on two major mobile messenger apps, WhatsApp and Facebook Messenger, our study also adds weight to previous evidence supporting the potential of mobile messenger applications in delivering vaccination knowledge, debunking vaccine-related misinformation, and providing vaccination guidance to unvaccinated individuals 53,54 . Our data showed that a majority of chatbot users were less likely to decrease in vaccine confidence and acceptance, compared with non-users; however, our study also found some apparent evidence of 'backfire' effects, specifically among some chatbot using parents in Hong Kong where we observed decreased vaccine acceptance with chatbot use. 'Backfire effects' are a controversial topic within the literature on digital health interventions-some research suggests that, in certain circumstances, pro-vaccine messaging delivered through social media can be counterproductive 29 . This might occur if, for example, messaging runs counter to the values of the target group or conflicts with individuals' personal experiences-in our case, we noted chatbots worked better for improving vaccine confidence and uptake among minority subpopulations and users with lower level of education, but were less effective in swaying the mainstream population with higher education levels. Some studies have also raised concerns that repeating misinformation in order to correct or debunk it can have the perverse effect of increasing people's familiarity with the misinformation, or may even spread misinformation to new audiences who had never been exposed to it before 55 . However, other studies have failed to replicate these findings 56 . In the case of our study, it is unclear why certain groups should have seen adverse outcomes on certain variables. Further in-depth investigations are needed. Conceivably, there may have been specific safety concerns or misinformation narratives that some had been less aware of prior to the study, and the process of engaging with the chatbot may have increased their familiarity with these topics or narratives.
Future chatbots may improve on their vaccine promotion and communication strategies as well as their message delivery, such as by using an emotion-based approach that can convey reassurances to chatbot users to ameliorate their doubts and fears 28,57 . An anthropomorphic chatbot that can share anecdotes may also have a positive impact; Loft et al. found that personal stories that go beyond facts and traditional sources of authority can be more persuasive in online communications campaigns 30 . Further advancement in AI technology, Natural Language Processing, and machine learning is immediately needed as the current chatbot operation relies heavily on human analysis to ensure response accuracy, especially in free text conversations. Further, chatbots should be supervised by trusted experts to ensure not only information accuracy, but data security and ethics compliance. Nevertheless, chatbots can be a useful component of a multi-pronged approach to health service delivery and communication, for example in combination with a webinar series or website with interactive features 29,58,59 . A more standardized assessment should be conducted to better analyse and improve chatbot's effectiveness in handling users' questions and influencing behaviours.
These study target populations who are unvaccinated or have delayed vaccination to identify viable strategies that could be applied in ongoing endeavours towards vaccine hesitancy alleviation 22,23,[60][61][62] . The findings of this study present that, while chatbots are a potentially beneficial intervention that could provide insights to policymakers on the nature of vaccine concerns and inform strategies that can better address vaccine hesitancy, evidence on the effectiveness of chatbots on vaccine confidence and uptake is not conclusive across different populations and requires further assessment. We suggest interventions be interpreted and modified to address idiosyncratic local contexts in order to reach optimal results. Concurrently, it is important that future investigations on chatbot interventions to enhance vaccine confidence include underrepresented minority groups, such as migrant workers, to broaden the applicability and scalability of chatbots.

Study design
We conducted a multisite, parallel, randomised controlled trial (RCT) designed to evaluate the effectiveness of COVID-19 chatbots for improving COVID-19 vaccine confidence and acceptance in three Asian locations concurrently: Thailand, Hong Kong, and Singapore. Randomisation and group allocation were performed by a participant recruitment and market research company, Premise. We ran a pilot study from January 17th to February 11th and validated the survey tools using confirmatory factor analysis (Supplementary Method 1) [63][64][65] . From February 11th to June 30th, 2022, eligible users in the online panel within Premise's mobile application were randomly invited to join the control or intervention group in our study with an allocation ratio of 1:1. The study was double-blinded-both participants and outcomes assessors were concealed from the intervention assignment. After signing the consent form, participants were asked to complete the pre-and post-intervention questionnaires and to use the chatbot if assigned to the intervention group. The formative, impact, and process assessment of the chatbot intervention was conducted following the RE-AIM framework 66,67 , and was quantitatively assessed based on its reach, efficacy, adoption, implementation, and maintenance.

Study population
Our study population included guardians of those who were unvaccinated or delayed their COVID-19 vaccinations until the government vaccine mandates (Supplementary Method 2 and Supplementary Fig. 1). Children and seniors had the lowest vaccination coverages in all study regions despite their COVID-19 disease vulnerability. Since guardians can make direct or indirect vaccination decision on behalf of children and seniors, we tested the effectiveness of chatbot in increasing guardians' vaccine confidence and acceptance for their dependent family members. Taking an average of estimates from similar studies conducted in Japan and France 53,68 , we estimated an effect size of 15% and determined a sample size of 250 for each of the control and intervention group using power analysis. However, our sample sizes did not reach the target due to the surge in vaccine uptake at the time of participant recruitment following the emergence of Omicron and subsequent governmental vaccine mandates, school-based vaccine rollouts and increased risk perceptions of COVID-19 ( Supplementary Fig. 1). In Thailand, the eligibility criteria included (1) adults with unvaccinated parents/grandparents aged 60 years or above, or (2) parents of unvaccinated children aged 5-11 years. The eligibility criteria in Hong Kong were slightly relaxed to include (1) adults whose parents/grandparents were 60 years old or above and had not been vaccinated against COVID-19 before the announcement of the mandatory Vaccine Pass on January 4, 2022 69 , or (2) parents of unvaccinated children under 18 years old. In Singapore, the eligibility criteria included parents of unvaccinated children aged 5-11 years (Supplementary Method 2). In all three locations, participants were recruited by Premise, a participant recruitment and market research company 70 , via random sampling using existing online panels.

Intervention
In Thailand, we adopted and updated ChatSure ( The COVID-19 vaccine chatbot content covered seven major categories of commonly asked questions: (1) Importance/Necessity, (2) Safety, (3) Effectiveness, (4) How to get vaccinated, (5) Tips before vaccination, (6) Tips after vaccination, and (7) Debunking COVID-19 vaccine-related misinformation. The information and dialogues were tailored and timely modified to the local contexts and regulations. Participants were allowed to enquire using English or the mainstream local languages (i.e., Thai in Thailand; Traditional Chinese, Simplified Chinese and English in Hong Kong; and Simplified Chinese and English in Singapore).
For both the intervention and control groups, pre-intervention surveys were sent to participants to determine their demographics. All participants were then asked to answer questions regarding COVID-19 vaccine confidence, including perceived importance, effectiveness, and safety, vaccine acceptance, and COVID-19 vaccine-related misinformation. Study instruments are available in Supplementary Methods 5-7.
Next, participants assigned to the intervention group were directed to use the chatbot for up to 1 week through an inviteonly link after completing the pre-intervention questionnaire. Participants in the intervention group were prompted by the chatbot to exchange at least ten messages regarding COVID-19 vaccines with the chatbot and submit screenshots of chatbot dialogues for validation before receiving a small reward for participation (<USD$15.00; Supplementary Table 12). Participants assigned to the control group were not given access to COVID-19 vaccine-related chatbots. After the intervention period, participants from both groups were asked to complete the same questionnaire about COVID-19 vaccines; chatbot users were asked to evaluate their usage experience.

Outcomes
The primary outcomes were changes in the levels of COVID-19 vaccine confidence and acceptance from the pre-and postintervention questionnaires. Confidence was measured using the Vaccine Confidence Index TM (VCI) 72 , which has been used in the context of COVID-19 vaccinations 73,74 and included perceptions of the importance, effectiveness, and safety of COVID-19 vaccines. VCI was recorded on a 5-point Likert scale (ranging from 5-strongly agree, 4-agree, 3-neither agree nor disagree, 2-disagree, to 1-strongly disagree) and the pre-and post-intervention differences were categorized into a 3-point scale of "improved", "no change", and "declined". For example, if a participant answered "Agree" in the pre-intervention questionnaire and "Strongly Agree" in the post-intervention questionnaire, the participant would score 1-point. Positive, zero, or negative differences were categorized into "improved", "no change", or "decreased" outcomes, respectively. Similarly, vaccine acceptance was measured by the change in participants' expressed willingness to vaccinate their senior parents/grandparents or children from the pre-and post-intervention questionnaires; the difference in pre-and postintervention responses was categorized into the same 3-point scale. Secondary outcomes included differences in perceived risks and benefits of COVID-19 vaccines and participants' awareness and knowledge of COVID-19 vaccine-related misinformation before and after the intervention. For the chatbot evaluation, outcomes were described based on the RE-AIM framework criteria (Supplementary Table 13), and conversation contents of D 2 4H chatbot and ChatSure in local languages were presented in word clouds ( Supplementary Figs. 4-7).

Data analysis
All participants who completed the assigned questionnaires and the intervention were analysed per protocol. The changes in outcome variables were measured on a 3-point scale ("improved", "no change", and "decreased") and compared by both Fisher's exact test and a proportional odds logistic regression model between the control and intervention groups to examine the effect of chatbot use on primary outcomes variables. We further employed proportional odds logistic regressions to investigate factors of primary outcome measures-vaccine confidence and acceptance where all participants' data were weighted with sex and ethnicity using the latest local census data 48,75,76 .

Reporting summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.

DATA AVAILABILITY
Anonymized data and code used can be found at: https://github.com/lkwok/ VCF_chatbot.